Fundamentals of Missing Data in Evaluation

Presentation to MSU Department of Psychology, Program Evaluation Brown Bag Series, East Lansing, MI

Steven J. Pierce

Center for Statistical Training and Consulting

2024-12-05

Outline

  • Problem
  • Ethics
  • Consequences
  • Diagnosis
  • Treatment
  • Prevention
  • Advice

Problem

Missing data (MD) is a common issue afflicting research and evaluation studies.

Warning

If you do much evaluation work, you will run into missing data.

What is Missing Data?

Any measurement you intended to obtain, but did not get.

  • Skipped items, instruments, visits, …
  • Illegible or irrelevant responses
  • Refusal to answer
  • Don’t know
  • Lost records
  • Study dropout

Ethics: Guiding Principles for Evaluators

Handling missing data well enacts our guiding principles[1]:

AEA logo.

  • Systematic inquiry
  • Competence
  • Integrity

Types of Missingness

  • Item-level
  • Construct
  • Person-period
  • Planned vs unplanned

Mechanisms of Missingness

  • Missing completely at random (MCAR)
  • Missing at random (MAR)
  • Missing not at random (MNAR)

MCAR

MCAR is when neither observed nor unobserved variables predict which data is missing

MAR

MNAR

Consequences

What are the consequences of missing data?

Bias

In appropriate handling of missing data can cause analyses to yield biased results.

Generalizability

Misalignment of analyzed sample and intended population

Power

Most statistical software defaults to listwise deletion of cases that have any missing values on the variables involved in an analysis. That reduces statistical

Diagnosis

Describing the Amount of MD

  • Numbers & percentages of complete & incomplete cases
  • Number and percentage of missing values for each variable
  • Nature and frequency of missing data patterns

Predictors of Attrition & Missingness in Longitudinal Studies

  • Study arm: Compare retention rates
  • Study site (in multisite studies)
  • Baseline/pretest values of outcome variables may predict who drops out or has missing values
  • Other covariates (demographics, site, )

Treatment

Prevention

An ounce of prevention is better than a pound of cure

[2]

Advice

  • Collaborate with a statistician!

Practical Options

  • Item-level missingness in scale scores[3,4]

References

1. American Evaluation Association. (2018). Guiding principles for evaluators [Web Page]. Author. https://www.eval.org/About/Guiding-Principles
2. Wisniewski, S. R., Leon, A. C., Otto, M. W., & Trivedi, M. H. (2006). Prevention of missing data in clinical research studies. Biological Psychiatry, 59, 997–1000. https://doi.org/10.1016/j.biopsych.2006.01.017
3. Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576. https://doi.org/10.1146/annurev.psych.58.110405.085530
4. Newman, D. A. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17(4), 372–411. https://doi.org/10.1177/1094428114548590

https://github.com/sjpierce/FMDE2024